PRETRAINED LANGUAGE MODELS

Buy Tesla, Sell Ford: Assessing Implicit Stock Market Preference in Pre-trained Language Models (2022.acl-short)

Challenge:	Pretrained language models do not account for the wide variety of available expert-generated language resources and lexicons that explicitly encode linguistic/domain knowledge.
Approach:	They propose a Python package for generating parallel representations for text based on predefined lexicons and selecting representations that provide the most information.
Outcome:	The proposed model can generate parallel representations of text based on predefined lexicons and select representations that provide the most information.

Copied to clipboard

Chengyu Chuang, Yi Yang

Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance (2025.naacl-industry)

Challenge:	Pretrained language models such as BERT have been used in many NLP tasks . however, there are still significant differences in their implicit preferences in the stock market .
Approach:	They assess the implicit stock market preferences in pretrained language models such as BERT . they find that there are significant differences in preferences between industry sectors .
Outcome:	The proposed model is more positive towards the stock market, but there are significant differences between industry sectors or within a sector.

Copied to clipboard

Somnath Banerjee, Avik Halder, Rajarshi Mandal, Sayan Layek, Ian Soboroff, Rima Hazra, Animesh Mukherjee

Data Contamination: From Memorization to Exploitation (2022.acl-short)

Challenge:	Pretrained language models (PLMs) have revolutionized NLP but amplify linguistic inequities in multilingual applications.
Approach:	They evaluate pretrained language models including Mistral, TowerInstruct, OpenHathi, Tamil-Llama, and Kan-Lama across eight languages spanning high-resource and low-resourced settings.
Outcome:	The proposed models fail to bridge linguistic divides and are inefficient when compared to other models.

Copied to clipboard

Inbal Magar, Roy Schwartz

Challenge:	Pretrained language models are typically trained on web-based datasets that are often "contaminated" with downstream test sets.
Approach:	They propose a method to pretrain BERT models on Wikipedia and labeled downstream datasets and fine-tune them on the relevant task.
Outcome:	The proposed method compares models on Wikipedia and labeled downstream datasets on two models and three downstream tasks.

Cooperative Self-training of Machine Reading Comprehension (2022.naacl-main)

Copied to clipboard

Hongyin Luo, Shang-Wen Li, Mingye Gao, Seunghak Yu, James Glass

Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models (2021.acl-short)

Challenge:	Pretrained language models provide high-quality contextualized word embeddings, but training question answering models requires large amounts of annotated data for specific domains.
Approach:	They propose a framework for automatically generating more non-trivial question-answer pairs to improve model performance.
Outcome:	The proposed framework outperforms state-of-the-art (SOTA) pretrained language models and transfer learning approaches on standard question-answering benchmarks.

Copied to clipboard

Mingyue Han, Yinglin Wang

Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)

Challenge:	Pretrained language models (PLMs) achieve surprising performance on the Choice of Plausible Alternatives (COPA) task.
Approach:	They propose to add a regularization loss to the existing COPA models to mitigate the problem of semantic similarity bias by adding a normalization loss.
Outcome:	The proposed model improves generalization ability and performs better on a challenging dataset, BCOPA-CE, which has unbiased token distribution and is more difficult for models to distinguish cause and effect.

Copied to clipboard

Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui

MIReAD: Simple Method for Learning High-quality Representations from Scientific Documents (2023.acl-short)

Challenge:	Pretrained language models have achieved remarkable success in various natural language processing tasks.
Approach:	They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance.
Outcome:	The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost.

Copied to clipboard

Anastasiia Razdaibiedina, Aleksandr Brechalov

The Web Can Be Your Oyster for Improving Language Models (2023.findings-acl)

Challenge:	Pretrained language models can learn rich textual representations, but they cannot provide powerful document-level representations for scientific articles.
Approach:	They propose a transformer-based method that learns semantically meaningful representations from scientific papers by fine-tuning transformer models to predict the target journal class based on the abstract.
Outcome:	The proposed method outperforms six existing models for representation learning on scientific documents across four evaluation standards.

Copied to clipboard

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jingyuan Wang, Jian-Yun Nie, Ji-Rong Wen

Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code (2025.coling-industry)

Challenge:	Pretrained language models encode a large amount of knowledge, but knowledge is frozen at the time of training, and the models become static and limited by training data.
Approach:	They propose an adaptive search engine assisted learning method that can self-evaluate the confidence level of PLM’s predictions and adaptively determine when to refer to the web for more data.
Outcome:	The proposed model outperforms retrieval-augmented methods on 16 knowledge-intensive tasks on a wide range of knowledge-related tasks.

Copied to clipboard

Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Barbosa Junior, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Nour Moustafa-Fahmy, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Hiep Nguyen, Sampo Pyysalo

Contrastive Demonstration Tuning for Pre-trained Language Models (2022.findings-emnlp)

Challenge:	Pretrained language models are integral part of AI applications, but their high computational cost limits accessibility.
Approach:	They evaluate Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code.
Outcome:	The proposed model outperforms existing models on English, Finnish, Hindi, Japanese, Vietnamese, and code.

Copied to clipboard

Xiaozhuan Liang, Ningyu Zhang, Siyuan Cheng, Zhenru Zhang, Chuanqi Tan, Huajun Chen

PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)

Challenge:	Recent studies focus on searching discrete or continuous prompts or optimized verbalizers, yet the demonstration examples are crucial for an excellent final performance of prompt-tuning.
Approach:	They propose a pluggable, extensible, and efficient approach to prompt tuning that is free of demonstration sampling.
Outcome:	The proposed approach can be pluggable, extensible, and efficient on 16 datasets.

Copied to clipboard

Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Kezhi Mao

A Self-supervised Joint Training Framework for Document Reranking (2022.findings-naacl)

Challenge:	Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens.
Approach:	They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations.
Outcome:	Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods.

Copied to clipboard

Xiaozhi Zhu, Tianyong Hao, Sijie Cheng, Fu Lee Wang, Hai Liu

Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)

Challenge:	Pretrained language models have been successfully applied to a wide range of tasks . however, the pretraining tasks were based on the context of documents .
Approach:	They propose a self-supervised joint training framework with a method called Masked Query Prediction to establish semantic relations between given queries and positive documents.
Outcome:	The proposed framework outperforms existing models on document reranking tasks without further pre-training . it uses a self-supervised method to establish semantic relations between given queries and positive documents.

Copied to clipboard

Timo Schick, Sahana Udupa, Hinrich Schütze

An Empirical Study on the Characteristics of Bias upon Context Length Variation for Bangla (2024.findings-acl)

Challenge:	Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet.
Approach:	They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text.
Outcome:	The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior.

Copied to clipboard

Jayanta Sadhu, Ayan Khan, Abhik Bhattacharjee, Rifat Shahriyar

CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment (2022.acl-short)

Challenge:	Language models exhibit various social biases due to widespread usage.
Approach:	They extend existing methods for measuring gender bias in Bangla by examining context length variation.
Outcome:	The proposed method relies on context length variation, highlighting the need for nuanced considerations in Bangla bias analysis.

Copied to clipboard

Lütfi Kerem Senel, Timo Schick, Hinrich Schuetze

Quantifying the Contextualization of Word Representations with Semantic Class Probing (2020.findings-emnlp)

Challenge:	Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks.
Approach:	They propose a benchmark that measures natural language understanding (NLU) abilities of pretrained language models.
Outcome:	The proposed benchmark measures the ability of pretrained language models to perform on many tasks.

Copied to clipboard

Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, Hinrich Schütze

Temporal Attention for Language Models (2022.findings-naacl)

Challenge:	Pretrained language models are effective in solving NLP tasks, but there are still questions about how and why they work so well.
Approach:	They use BERT to quantify contextualization by studying the extent of inference . they show that top layer representations support highly accurate inference of semantic classes .
Outcome:	The proposed model is highly accurate, but weak in the lower layers . it is more task-specific after finetuning while lower layers are more transferable .

Copied to clipboard

Guy D. Rosin, Kira Radinsky

A Study of Implicit Bias in Pretrained Language Models against People with Disabilities (2022.coling-1)

Challenge:	Pretrained language models are trained on corpora derived from the web, but ignore this information.
Approach:	They propose a time-aware self-attention mechanism that captures time-specific contextualized word representations and allows the transformer to capture this information.
Outcome:	The proposed model achieves state-of-the-art on three datasets in different languages (English, German, and Latin) that vary in time, size, and genre.

Copied to clipboard

Pranav Narayanan Venkit, Mukund Srinath, Shomir Wilson

The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models (2023.eacl-main)

Challenge:	Pretrained language models exhibit sociodemographic biases, such as against gender and race, raising concerns of downstream biase in language technologies.
Approach:	They propose to use word embedding-based and transformer-based PLMs to test for the presence of biases against people with disabilities (PWDs)
Outcome:	The proposed models favor ableist language, despite their sociodemographic biases against race and gender.

Copied to clipboard

Son Quoc Tran, Phong Nguyen-Thuan Do, Uyen Le, Matt Kretchmar

Robustness in NLP

Challenge:	Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks.
Approach:	They propose to fine-tune three state-of-the-art language models on SQuAD 1.1 or SQu AD 2.0 and then evaluate their robustness under adversarial attacks.
Outcome:	The proposed model is able to perform better under adversarial attacks than model fine-tuned on SQuAD 1.1 or 2.0.

Better Robustness by More Coverage: Adversarial and Mixup Data Augmentation for Robust Finetuning (2021.findings-acl)

Copied to clipboard

Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, Maosong Sun

AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models (2023.findings-eacl)

Challenge:	Pretrained language models perform poorly under adversarial attacks due to the large search space.
Approach:	They propose a method to cover a much larger proportion of the attack search space by adding textual adversarial examples during training.
Outcome:	The proposed method covers a much larger proportion of the attack search space.

Copied to clipboard

Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, Jesse Dodge

Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2021.eacl-main)

Challenge:	Pretrained language models often need to specialize to specific domains.
Approach:	They propose an approach that performs weight-space averaging of adapters trained on different domains.
Outcome:	The proposed approach improves performance to new domains without extra training.

Copied to clipboard

Benjamin Heinzerling, Kentaro Inui

CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020.emnlp-main)

Challenge:	Pretrained language models have been suggested as an alternative or complement to structured knowledge bases . however, this paradigm has only been considered in a very limited setting .
Approach:	They propose a paradigm that allows LMs to store a large number of entities . they propose LM-as-KB paradigm which allows querying stored facts .
Outcome:	The proposed paradigm allows handling 21k entities whose name is found in common LM vocabularies . the proposed paradigm has only been considered in a very limited setting .

Copied to clipboard

Nikita Nangia, Clara Vania, Rasika Bhalerao, Samuel R. Bowman

DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models (2023.emnlp-main)

Challenge:	Pretrained language models use cultural biases implicitly, causing harm . identifying and quantifying learnt biase enables us to measure progress .
Approach:	They propose a benchmark to measure social bias in pretrained language models . they use 1508 examples that cover stereotypes dealing with nine types of bias .
Outcome:	The proposed benchmark focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups.

Copied to clipboard

Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, Deyi Xiong

Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models (2023.eacl-main)

Challenge:	Existing studies have demonstrated that pretrained language models memorize and regurgitate a significant portion of training data, including atypical data points that appear only once in the training data.
Approach:	They propose a method to locate and erase risky neurons in order to eliminate the impact of privacy data in the model in batches.
Outcome:	The proposed method eliminates the impact of privacy data in the model in batches without affecting the model's performance.

Copied to clipboard

Zdeněk Kasner, Ioannis Konstas, Ondrej Dusek

Quadapter: Adapter for GPT-2 Quantization (2022.findings-emnlp)

Challenge:	Pretrained language models (PLMs) for data-to-text generation produce inaccurate outputs if labels are ambiguous or incomplete, which is often the case in D2T datasets.
Approach:	They propose to use a dataset to descib a relation between two entities using relation labels to train pretrained language models.
Outcome:	The proposed models are robust to generalizing to out-of-domain domains on a dataset for descibing a relation between two entities.

Copied to clipboard

Minseop Park, Jaeseong You, Markus Nagel, Simyung Chang

Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples (2023.findings-emnlp)

Challenge:	Quantization-aware training (QAT) fine-tunes model parameters in response to the numerical error arising from quantization.
Approach:	They propose a quantization adapter that scales activations channel-wise to make activations quantization-friendly by scaling them channel- wise.
Outcome:	The proposed method prevents overfitting and improves quantization performance.

Copied to clipboard

Zixuan Ren, Yang Zhao, Chengqing Zong

Probing Pre-trained Language Models for Semantic Attributes and their Values (2021.findings-emnlp)

Challenge:	Pretrained language models (PLMs) have impressive capabilities in open-ended text generation.
Approach:	They propose a dynamic knowledge-guided informative open-ended text generation approach that utilizes a knowledge graph to help the model generate more contextually related entities and detailed facts.
Outcome:	The proposed approach generates more informative texts than baselines.

Copied to clipboard

Meriem Beloucif, Chris Biemann

LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization (2024.findings-emnlp)

Challenge:	Pretrained language models (PTLMs) are used for many tasks including syntax, semantics and commonsense.
Approach:	They propose to integrate semantic attributes and their values into pretrained language models to improve their performance on many natural language processing tasks.
Outcome:	The proposed model performs better on masked tokens than humans on this task.

Copied to clipboard

Muhammad Adilazuarda, Samuel Cahyawijaya, Genta Winata, Ayu Purwarianti, Alham Aji

Multilinguality

Challenge:	Pretrained language models have shown remarkable generalization toward multiple tasks and languages, but their generalization towards unseen languages is poor.
Approach:	They propose a regularization technique that incorporates various aspects of languages to better characterize linguistics constraints.
Outcome:	The proposed technique improves accuracy of mBERT and XLM-R on unseen languages by 18% and 2% compared to fully finetuned models.

Extremely Small BERT Models from Mixed-Vocabulary Training (2021.eacl-main)

Copied to clipboard

Sanqiang Zhao, Raghav Gupta, Yang Song, Denny Zhou

Factual Consistency of Multilingual Pretrained Language Models (2022.findings-acl)

Challenge:	Existing knowledge distillation methods cannot be directly applied to train student models with reduced vocabulary and embedding dimensions.
Approach:	They propose a method to align teacher and student embeddings via mixed-vocabulary training.
Outcome:	The proposed method compresses BERT-LARGE to a task-agnostic model with smaller vocabulary and hidden dimensions, which is an order of magnitude smaller than other distilled models.

Copied to clipboard

Constanza Fierro, Anders Søgaard

Task-guided Disentangled Tuning for Pretrained Language Models (2022.findings-acl)

Challenge:	Recent work shows that monolingual English language models fill-in-the-blank differently for paraphrases describing the same fact.
Approach:	They propose a resource to analyze consistency of English language models . they find that mBERT is as inconsistent as English BERT in paraphrases .
Outcome:	The proposed model is as inconsistent as English BERT in English paraphrases, but it is more so for all the other 45 languages.

Copied to clipboard

Jiali Zeng, Yufan Jiang, Shuangzhi Wu, Yongjing Yin, Mu Li

Lifting the Curse of Capacity Gap in Distilling Language Models (2023.acl-long)

Challenge:	Pretrained language models are fine-tuned on task-specific datasets, but fail to capture task- specific patterns.
Approach:	They propose a method which disentangles task-relevant signals from entangled representations.
Outcome:	The proposed method improves generalization of representations by disentangling task-relevant signals from the entangled representations.

Copied to clipboard

Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, Dawei Song

End-to-end Case-Based Reasoning for Commonsense Knowledge Base Completion (2023.eacl-main)

Challenge:	Existing studies have shown that pretrained language models require a tremendous amount of inference compute to perform.
Approach:	They propose to compress pretrained language models to small ones with a teacher-student paradigm to fill the capacity gap.
Outcome:	The proposed model achieves state-of-the-art performance at small FLOPs compared with competitive baselines.

Copied to clipboard

Zonglin Yang, Xinya Du, Erik Cambria, Claire Cardie

Learning to Transfer Prompts for Text Generation (2022.naacl-main)

Challenge:	Pretrained language models have been shown to store knowledge in their parameters and have achieved reasonable performance in knowledge-intensive tasks.
Approach:	They propose to provide retrieved passages that contain relevant knowledge as additional input to the commonsense knowledge base completion (CKBC) task.
Outcome:	The proposed framework generates more valid, informative, and novel knowledge than the state-of-the-art COMET model for commonsense knowledge base completion (CKBC) tasks.

Copied to clipboard

Junyi Li, Tianyi Tang, Jian-Yun Nie, Ji-Rong Wen, Xin Zhao

Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization (2023.findings-emnlp)

Challenge:	Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning.
Approach:	They propose a prompt-based method that learns source prompts and transfers them as target prompts to perform target generation tasks.
Outcome:	The proposed method can be used to perform text generation tasks in a transferable setting.

Copied to clipboard

Shoujie Tong, Heming Xia, Damai Dai, Runxin Xu, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui

Structural Adapters in Pretrained Language Models for AMR-to-Text Generation (2021.emnlp-main)

Challenge:	Pretrained language models can be fine-tuned on limited training data, which can overfit and thus diminish performance.
Approach:	They propose a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout.
Outcome:	The proposed method outperforms existing methods on the GLUE benchmark and exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios.

Copied to clipboard

Leonardo F. R. Ribeiro, Yue Zhang, Iryna Gurevych

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2022.naacl-main)

Challenge:	Pretrained language models (PLMs) have advanced graph-to-text generation, but efficient encoding of graph structure is challenging because of the nature of the data.
Approach:	They propose a method to encode graph structure into pretrained language models by training only graph structure-aware adapter parameters.
Outcome:	The proposed method outperforms the state-of-the-art on two AMR-to-text datasets, training only 5.1% of the adapter parameters.

Copied to clipboard

Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren

Discovering Knowledge-Critical Subnetworks in Pretrained Language Models (2024.emnlp-main)

Challenge:	Pretrained language models are typically learned over a large, static corpus and fine-tuned for various downstream tasks.
Approach:	They propose to continuously update a pretrained language model to adapt to emerging data and to keep track of the model's performance.
Outcome:	The proposed model can adapt to new corpora while retaining knowledge in earlier domains.

Copied to clipboard

Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, Antoine Bosselut

Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations (2020.coling-main)

Challenge:	Pretrained language models encode implicit representations of knowledge in their parameters, but localizing these representations and disentangling them from each other remains an open problem.
Approach:	They propose a masking scheme that can be applied to weights and neurons to discover such subnetworks.
Outcome:	The proposed method can remove specific knowledge from models while minimizing adverse effects on the original model.

Copied to clipboard

Sheng Liang, Philipp Dufter, Hinrich Schütze

DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoising (2025.acl-long)

Challenge:	Prior work identifies a linear gender subspace and removes gender information by eliminating the subspace.
Approach:	They propose to use DensRay to obtain interpretable dense subspaces by applying it to attention heads and layers of BERT.
Outcome:	The proposed method performs on-par with prior approaches, but is more robust and preserves language model performance better.

Copied to clipboard

Zhenhao Li, Huichi Zhou, Marek Rei, Lucia Specia

Robustness in NLP

Challenge:	Existing adversarial defense methods for natural language processing still pose challenges to adversarials.
Approach:	They propose a novel adversarial defense method that incorporates a diffusion layer as a denoiser between the encoder and the classifier.
Outcome:	The proposed method improves over existing adversarial defense methods and achieves state-of-the-art performance against black-box and white-box adversarials.

Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment (2023.findings-emnlp)

Copied to clipboard

Boyang Xue, Weichao Wang, Hongru Wang, Fei Mi, Rui Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong

Enhancing Hindi Feature Representation through Fusion of Dual-Script Word Embeddings (2024.lrec-main)

Challenge:	Experimental results show that pretrained language models generate inconsistent factual knowledge in many conversational tasks.
Approach:	They propose a method which explicitly introduces extended feedforward networks (FFNs) in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs.
Outcome:	The proposed methods improve the factual expression capability of feedforward networks (FFNs) in knowledge-grounded dialogue systems by knowledge enhancement and alignment respectively.

Copied to clipboard

Lianxi Wang, Yujia Tian, Zhuowei Chen

MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)

Challenge:	Pretrained language models often neglect the integration of different scripts within a language, constraining their ability to capture richer semantic information.
Approach:	They propose a dual-script enhanced feature representation method for Hindi . they combine features from Devanagari and Romanized Hindi Roberta .
Outcome:	The proposed method improves model performance across multiple natural language processing tasks.

Copied to clipboard

Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, Jeremy Howard

Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models (2024.lrec-main)

Challenge:	Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages.
Approach:	They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language.
Outcome:	The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks.

Copied to clipboard

Furkan Akkurt, Onur Gungor, Büşra Marşan, Tunga Gungor, Balkiz Ozturk Basaran, Arzucan Özgür, Susan Uskudarli

Incremental Neural Lexical Coherence Modeling (2020.coling-main)

Challenge:	Pretrained language models and large language models are increasingly used to assist in a variety of natural language processing tasks.
Approach:	They propose to use pretrained language models and large language models to evaluate their quality in natural language processing.
Outcome:	The proposed annotation scheme (2.11) yields sentences with higher success rate than the previous one.

Copied to clipboard

Sungho Jeon, Michael Strube

Designing Informative Metrics for Few-Shot Example Selection (2024.findings-acl)

Challenge:	Recent work on pretrained language models has led to significant improvements in a range of NLP tasks.
Approach:	They propose a coherence model which interprets sentences incrementally to capture lexical relations between them.
Outcome:	The proposed model interprets sentences incrementally to capture lexical relations between them.

Copied to clipboard

Rishabh Adiga, Lakshmi Subramanian, Varun Chandrasekaran

Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models (2024.findings-emnlp)

Challenge:	Pretrained language models (PLMs) have shown remarkable few-shot learning capabilities when provided with properly formatted examples.
Approach:	They propose a complexity-based prompt selection approach for sequence tagging tasks that uses certain metrics to align the syntactico-semantic complexity of test sentences and examples.
Outcome:	The proposed approach achieves state-of-the-art performance on few-shot NER, with 5% improvement in F1 score.

Copied to clipboard

Minseok Choi, Kyunghyun Min, Jaegul Choo

CamemBERT: a Tasty French Language Model (2020.acl-main)

Challenge:	Pretrained language models memorize large amounts of information, raising significant safety concerns.
Approach:	They propose an approach to machine unlearning for multilingual language models that selectively erases information across different languages while maintaining overall performance.
Outcome:	The proposed approach is compared with existing unlearning baselines and set a new standard for secure and adaptable multilingual language models.

Copied to clipboard

Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, Benoît Sagot

Gender Identity in Pretrained Language Models: An Inclusive Approach to Data Creation and Probing (2024.findings-emnlp)

Challenge:	Pretrained language models are now ubiquitous in Natural Language Processing, but their use in other languages is limited.
Approach:	They propose to train monolingual Transformer-based model for other languages using web crawled data instead of Wikipedia data and a relatively small web crawl dataset leads to better results.
Outcome:	The proposed model performs as well as those obtained using larger datasets.

Copied to clipboard

Urban Knupleš, Agnieszka Falenska, Filip Miletić

Implicit Representations of Grammaticality in Language Models (2026.acl-long)

Challenge:	Pretrained language models encode binary gender information of text authors, raising the risk of skewed representations and downstream harms.
Approach:	They use a corpus of YouTube transcripts from transgender, cisgender and non-binary speakers to examine whether pretrained language models encode binary gender information.
Outcome:	The proposed model encodes gender information for all gender identities but to different extents.

Copied to clipboard

Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim

Language Models are Few-Shot Butlers (2021.emnlp-main)

Challenge:	Pretrained language models generate grammatically well-formed text and discriminate well between grammatical and ungrammatically sentences in tightly controlled minimal pairs.
Approach:	They propose a method to train pretrained LMs for representations of grammaticality by applying perturbations to a naturalistic text corpus.
Outcome:	The proposed model outperforms probability-based models on human-curated grammaticality judgment benchmarks and performs worse than string probabilities on plausibility benchmarks.

Copied to clipboard

Vincent Micheli, Francois Fleuret

Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control (2023.findings-acl)

Challenge:	Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets.
Approach:	They propose a two-stage procedure to learn from a small set of demonstrations and a simple reinforcement learning algorithm to improve by interacting with an environment.
Outcome:	The proposed method improves with only 1.2% of the demonstrations and a simple reinforcement learning algorithm over existing methods in the ALFWorld environment.

Copied to clipboard

Xiang Fan, Yiwei Lyu, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency

SPE: Symmetrical Prompt Enhancement for Fact Probing (2022.emnlp-main)

Challenge:	Existing methods for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.
Approach:	They propose a few-shot human-in-the-loop training algorithm that allows distribution control for text generation via human feedback.
Outcome:	The proposed algorithm achieves state-of-the-art results on single topic/attribute and quantified distribution control compared to previous works.

Copied to clipboard

Yiyuan Li, Tong Che, Yezhen Wang, Zhengbao Jiang, Caiming Xiong, Snigdha Chaturvedi

Context Minimization for Resource-Constrained Text Classification: Optimizing Performance-Efficiency Trade-offs through Linguistic Features (2025.findings-emnlp)

Challenge:	Recent work probes PLMs for the extent of factual knowledge through prompts . however, these methods do not consider symmetry of the task: object and subject prediction.
Approach:	They propose a continuous prompt-based method that leverages symmetry of the task by constructing symmetrical prompts for subject and object prediction.
Outcome:	The proposed method improves on a popular factual probing dataset on lAMA.

Copied to clipboard

Nahid Hossain, Md Faisal Kabir

Information Retrieval

Challenge:	Pretrained language models have transformed text classification, but their computational demands often render them impractical for resource-constrained settings.
Approach:	They propose a linguistically-grounded framework for context minimization that leverages theme-rheme structure to preserve critical classification signals while reducing input complexity.
Outcome:	The proposed framework preserves critical classification signals while reducing input complexity.

Language-Agnostic Bias Detection in Language Models with Bias Probing (2023.findings-emnlp)

Copied to clipboard

Abdullatif Köksal, Omer Yalcin, Ahmet Akbiyik, M. Kilavuz, Anna Korhonen, Hinrich Schuetze

The Benefits of Label-Description Training for Zero-Shot Text Classification (2023.emnlp-main)

Challenge:	Pretrained language models (PLMs) contain strong social biases, which are difficult to quantify because current methods focusing on fill-the-mask objectives are sensitive to slight changes in input.
Approach:	They propose a bias probing technique called LABDet to evaluate social bias in pretrained language models with a language-agnostic method.
Outcome:	The proposed method “surfaces” nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection.

Copied to clipboard

Lingyu Gao, Debanjan Ghosh, Kevin Gimpel

A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives (2024.emnlp-main)

Challenge:	Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data to classify among specific label sets in downstream tasks.
Approach:	They propose to use a small finetuning dataset to describe the labels for a task and to use it to further improve zero-shot accuracies.
Outcome:	The proposed model is more accurate than zero-shot by 17-19% absolute across topic and sentiment datasets and more robust to choices required for zero- shot classification.

Copied to clipboard

Zihao Li, Shaoxiong Ji, Timothee Mickus, Vincent Segonne, Jörg Tiedemann

Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Models (2023.findings-emnlp)

Challenge:	Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community.
Approach:	They propose to compare multilingual pretraining objectives in a controlled methodological environment with multilingual models.
Outcome:	The proposed model outperforms existing models in 6 languages and demonstrates that multilingual translation is an effective pretraining objective under the right conditions.

Copied to clipboard

You Li, Jinhui Yin, Yuming Lin

Language models and brains align due to more than next-word prediction and word-level information (2024.emnlp-main)

Challenge:	Pretrained language models are expected to map input text to a set of vectors while preserving the inherent relationships within the text.
Approach:	They propose a new line to constructing metrics for understanding the mechanisms of pretrained language models.
Outcome:	The proposed model is referred to as the tree topological probe and is based on a tree topology probe.

Copied to clipboard

Gabriele Merlin, Mariya Toneva

ORTicket: Let One Robust BERT Ticket Transfer across Different Tasks (2024.lrec-main)

Challenge:	Pretrained language models have been shown to significantly predict brain recordings of people comprehending language.
Approach:	They propose to use two perturbations to design contrasts that control for different types of information.
Outcome:	The proposed model is largely agnostic about the exact linguistic information contained in the conceptual quantities "word-level information" and "multi-word information".

Copied to clipboard

Yuhao Zhou, Wenxiang Chen, Rui Zheng, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)

Challenge:	Pretrained language models are susceptible to subtle perturbations and require multiple adversarial training during fine-tuning to improve their robustness.
Approach:	They propose a novel adversarial defense method ORTicket that fine-tunes a model for downstream tasks.
Outcome:	The proposed method achieves comparable robustness to other defense methods while maintaining the efficiency of fine-tuning.

Copied to clipboard

Marek Kadlčík, Michal Štefánik, Timothee Mickus, Josef Kuchař, Michal Spiegel